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We calculate with Monte Carlo the goodness of fit and the confidence level of the standard allowed regions for 
the neutrino oscillation parameters obtained from the fit of the total rates measured in solar neutrino experiments. 
We show that they are significantly overestimated in the standard method. We also calculate exact allowed regions 
with correct frequentist coverage. We show that the exact VO, LMA and LOW regions are much larger than the 
standard ones and merge together giving an allowed band at large mixing angles for all Am 2 > 10 -10 eV 2 . 



1. Introduction 

The data of solar neutrino experiments provide 
strong indications in favor of neutrino oscillations 
and their statistical analysis give indications on 
the values of the neutrino mixing parameters. For 
two neutrino generations, these parameters are 
the mass-squared difference Am 2 = m\ — mf, 
where mi and m<i are the two neutrino masses, 
and tan 2 9, where 9 is the mixing angle (see, for 
example, Ref.0). 

In this paper we present statistical methods 
based on Monte Carlo numerical calculations that 
allow to improve the standard statistical analy- 
sis of solar neutrino data, which is approximate 
j2|. We consider the data relative to the to- 
tal rates measured in the Homestake and Super- 
Kamiokande experiments, and the weighted aver- 
age of the total rates measured in the two Gal- 
lium experiments GALLEX and SAGE, as given 
in Table I of Ref.§. 

As in the standard method, we use as estimator 
of Am 2 and tan 2 6 the global minimum X^ in of 
the least-squares function 
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where iV CX p = 3 is the number of experimental 
data points, V is the covariance matrix of experi- 
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mental and theoretical uncertainties, R^ exp ^ is the 
event rate measured in the j th experiment and 
i?j thr ' ) is the corresponding theoretical event rate, 
that depends on Am 2 and tan 2 9. We calculate 
the covariance matrix V following the standard 
method (see, for example, Ref. ||), with the cor- 
rection proposed in Ref.Q]. 

The standard procedure to calculate the al- 
lowed regions in the neutrino oscillation parame- 
ter space (see, for example, Ref.||) is based on 
the assumption that X 2 has a x 2 distribution 
with A^ eX p degrees of freedom. This would be cor- 
rect if the theoretical rates depended lin- 
early on the parameters to be determined in the 
fit and the errors were multinor- 
mally distributed with constant covariance ma- 
trix V . In this case there should be only one mini- 
mum of X 2 and the allowed regions at 100/3% con- 
fidence level (CL), given by X 2 < Xl in +AX 2 {(3), 
should be elliptical. 

It is well-known that in reality there are sev- 
eral local minima of X 2 , each one determining an 
allowed region, and these allowed regions do not 
have elliptic form (see, for example, Ref.||). This 
is due to the fact that the requirements above are 
not satisfied. In particular, the stronger effect 
is due to the non-linear dependence of the the- 
oretical rates i?j thr ' ) from the parameters, which 
generates several local minima of X 2 . 

In the following two sections we present esti- 
mations of the goodness of fit (Section ||) and the 
confidence level of the standard allowed regions 
(Section ||) obtained with a Monte Carlo calcu- 
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lation of the distribution of X 2 . In Section || we 
present the results of a Monte Carlo calculation 
of allowed regions with exact coverage. 

2. Goodness of Fit 

Goodness of fit (GOF) is the probability to find 
a value of the global minimum X 2 nin of X 2 larger 
than the one obtained from the fit. Since there are 
more than one local minima of X 2 with relatively 
close values of X 2 , there are more possibilities 
to obtain good fits of the data with respect to 
the case of one minimum, and the true goodness 
of fit is smaller than the one obtained with the 
standard method. 

We calculate the distribution of X^ in assum- 
ing that the best-fit values Am 2 , tan 2 9 of Am 2 , 
tan 2 9 are reasonable surrogates of the true un- 
known values Am 2 ruc , tan 2 #true and the probabil- 
ity distribution of the differences Aro 2 ^) — Am 2 , 
tan 2 6rty — tan 2 6 is not too different from the true 
distribution of the differences Am 2 ^ — Am 2 rue , 
tan 2 Off.) ~ tan 2 9 tIUC in a large set of best-fit pa- 
rameters Am 2 (fc), tan 2 9^ (k = 1,2,...) ob- 
tained with hypothetical experiments. 

Using Am 2 , tan 2 9 as surrogates of the true val- 
ues, we generate N s synthetic random data sets 
with the usual gaussian distribution for the exper- 
imental and theoretical uncertainties. We apply 
the least-squares method to each synthetic data 
set, leading to an ensemble of simulated best-fit 

parameters Ara 2 ( s ), tan 2 9^ with s — 1, . . . , N s , 
each one with his associated (AT^Js. Then we 
calculate the goodness of the fit as the fraction of 
simulated (X^ in ) s in the ensemble that arc larger 
than the one actually observed, A^ in . 

The global minimum of the least-squares func- 
tion (0), X^ in = 0.42, occurs in the SMA regionR 
for Am 2 = 5.1xlCT 6 eV 2 and tan 2 9 = 1.6 xlCT^ 
The Monte Carlo method yields a GOF of 40%, 
that must be compared with the 52% GOF ob- 



2 We use the standard terminology for the allowed regions 
(see, for example, Ref.E)): SMA for Am 2 ~5x 10" 6 eV 2 , 
tan 2 9 ~ 1CT 3 , LMA for Am 2 ~ 3 x fCT 5 eV 2 , tan 2 8 ~ 
0.3, LOW for Am 2 ~ 10 _7 eV 2 , tan 2 9 ~ 0.5, VO for 
Am 2 < 10" 8 eV 2 . 



taincd with the standard method. Therefore, the 
standard method significantly overestimates the 
GOF. 

3. Confidence Level of Allowed Regions 

The allowed regions with 100/3% CL are defined 
by the property that they belong to a set of al- 
lowed regions, obtained with hypothetical experi- 
ments, which cover (i.e. include) the true value of 
the parameters with probability (3. This property 
is called coverage. 

When there are several local minima of X 2 with 
relatively close values of X 2 , in repeated experi- 
ments the global minimum has significant chances 
to occur far from the true (unknown) value of the 
parameters, leading to a smaller probability that 
the allowed regions cover the true value with re- 
spect to the case in which there is only one min- 
imum, which is assumed for the validity of the 
standard method for the calculation of allowed 
regions. Hence, the true confidence level of a stan- 
dard 100/3% CL allowed region is smaller than (3. 

We estimate the true confidence level of the 
standard 100/3% CL allowed regions using the 

best fit values of the parameters, Am 2 , tan 2 9, as 
surrogates of the true values, Am 2 rue , tan 2 6* trU c, 
for the generation of a large number of synthetic 
data sets. We apply the standard procedure to 
each synthetic data set and obtain the corre- 
sponding standard 100/3% CL allowed regions in 
the space of the neutrino oscillation parameters. 
Then we count the number of synthetic standard 
100/3% CL allowed regions that cover the assumed 

surrogate Am 2 , tan 2 9 of the true values. The ra- 
tio of this number and the total number of syn- 
thetically generated data set gives a Monte Carlo 
estimation /3mc of the true confidence level of the 
standard 100/3% CL allowed regions. 

We obtained that the confidence level of the 
standard 90% CL allowed regions is 86%, which is 
significantly smaller. Other results are presented 
in Ref.§. 

4. Exact Allowed Regions 

The calculation of the confidence level of the 
standard allowed regions presented in the previ- 
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Figure 1. Allowed 90%, 95%, 99%, 99.73% CL regions. The gray areas are the allowed regions with 
exact frequentist coverage. The areas enclosed by the solid lines are the standard allowed regions. 



ous section is approximate, because it is based on 
the assumption of a surrogate for the unknown 
true values of the neutrino oscillation parameters. 
It would be useful to be able to calculate exact al- 
lowed with the desired confidence level, i.e. with 
correct coverage. The procedure that allows to 
perform this task has been invented by Neyman 
in 1937 (see references in Ref. Q). Let us empha- 
size that this procedure gives allowed regions with 
proper coverage for any unknown true values of 
the parameters. 

Our implementation of Neyman's procedure 
for the calculation of the allowed regions in 
the tan 2 8- Am 2 plane is described in details in 
Ref. H . The results are presented in Fig.|l], where 
the gray areas are the exact allowed regions, and 
the standard allowed regions are enclosed by solid 
lines. One can see that the exact LMA, LOW and 
VO regions are much larger than the standard 
ones and there is no separation between them. 
Hence, large mixing angles with 0.2 < tan 2 9 < 1 
are allowed for Am 2 > 10 _10 eV 2 . On the other 
hand, the exact SMA region approximately coin- 
cides with the standard one. This is due to the 
fact that in the SMA region the assumption of a 



in Eq. ([j]) from the parameters tan 2 9, Am 2 is ap- 
proximately correct, whereas it is violated quite 
badly in the LMA, LOW and VO regions. 



5. Conclusions 

In conclusion, we have shown that the stan- 
dard method used in the analysis of solar neu- 
trino data in terms of neutrino oscillations signif- 
icantly overestimates the goodness of fit and the 
confidence level of the allowed regions. We have 
also calculated allowed regions with correct cov- 
erage. The SMA region approximately coincides 
with the standard one, but the VO, LMA and 
LOW regions are much larger than the standard 
ones and merge together giving an allowed band 
at large mixing angles for all Am 2 > 10~ 10 eV 2 . 
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